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ABSTRACT 

While mail surveys offer an efficient and cost 
effective means of gathering large amounts of student-related data in 
a relatively sho^t time, surveys characterized by low response rates 
may have a biasing effect on information. To examine this bias, a 
study was undertaken of the characteristics associated with samples 
from three graduate follow-up surveys conducted at CCP between 1989 
and 1991- Demographic characteristics, entering academic abilities, 
academic engagement, and persistence were examined for survey 
respondents, voluntary nonrespondents who received the survey and 
chose not to participate, and involuntary nonrespondents who did not 
receive the questionnaire due to an incorrect address. The most 
consistent gi oup differences were associated with academic 
performance and persistence. Respondents earned higher grade point 
averages (GPA's) and persisted for a longer time than nonrespondents. 
Voluntary nonrespondents were less academically successful and 
persisted for longer periods than involuntary nonrespondents. Results 
indicate that survey results may be skewed since the more positive a 
student's 3 ademic experience, the more likely she/he is to respond. 
The following strategies are suggested to reduce the number of 
nonresponses: (1) send a short, professional introductory letter and 
a simple, logical questionnaire; (2) include a self-addressed stamped 
envelope; (3) personalize the letter; (4) send a second survey to 
nonrespondents; and (5) remind students of the importance of making 
address changes. Contains 19 references. (KP) 
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Executive Summary 
This study explored characteristics associated with 
respondents and nonrespondents to mail surveys in order to 
better understand the extent and nature of nonresponse bias 
associated with information gathered through this method. 
Three groups of former students were included in comparative 
analyses: 1) survey respondents, 2) voluntary nonrespondents, 
and 3) involuntary nonrespondents. The latter two groups 
were distinguished from each other based on receipt of the 
survey. The second group received the questionnaire and 
chose not to respond while for the third group nonresponse 
was the result of not receiving the questionnaire since CCP 
did not have a current mailing address for the former 
student . 

Data were obtained from three follow-up surveys of 
former students who either graduated or left the College 
prior to graduating. Possible biasing factors in the study 
included measures of demographic characteristics, entering 
academic abilities, academic experiences, and persistence. 

In general, respondents tended to be more dissimilar to 
the two nonrespondent groups than the nonrespondent groups 
were to each other. Across the three surveys the most 
consistent group differences were associated with academic 
performance and persistence. Respondents earned higher GPAs 
and persisted for a longer time at the College than 
nonrespondents. Voluntary nonrespondents were less 
academically successful and persisted for longer periods of 
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time than involuntary nonrespondents . To a lesser degree, 
group differences were also found based on age and ethnicity* 

These results indicate that survey results, when 
characterized by low response rates, need to be used 
cautiously if generalized to students who do not participate 
in the survey. Survey results could be positively skewed 
since it appears that the more positive, the students' 
academic experiences at the College, the more likely students 
are to respond. 



Introduction 

Mail surveys are frequently used by colleges and 
universities to gather information about the post-college 
experiences of their forrcer students. While mail survey 
methods offer an efficient and cost effective means of 
gathering large amounts of student-related data in a 
relatively short period of time, there are disadvantages 
associated with these methods. Mail surveys can be 
characterized by low response rates which may have a biasing 
effect on information gathered in this manner. Unsuitably 
low response rates to mail questionnaires and the consequent 
self -selected nature of the sample on which these survey 
findings are based lead to skepticism regarding the 
representativeness of respondent information. Self -selection 
of respondents can make inferences drawn from the 
questionnaire data tenuous and minimise the perceived 
usefulness of the questionnaire data (Berdie & Anderson, 
1974; Fawler, 1984; Hutchinson et al., 1987; Smith & Bers, 
1987) . 

A typical strategy that is used to establish the amount 
of confidence that can be placed in the general j ^ability of 
information gathered through these methods is to compare the 
profiles of survey respondents with nonrespondents or late 
respondents to determine if the groups differ with regard to 
contrasted characteristics (Hutchinson et al., 1987; Carifio 
et al., 1991). If these comparative analyses result in 



nonsignificant differences, survey results are generally 
treated as fair representations of nonresponding students. 
If, on the other hand, the two groups are determined to be 
significantly different with regard to contrasted 
characteristics, survey results are either disregarded or 
used cautiously after applying weighting procedures that 
attempt to compensate for the possible nonresponse bias in 
the sample (Astin & Panos, 1969; Astin, 1970; Grosset & 
Boris, 1982; Kathon, 1983; Hutchinson et al., 1987; 
Goodfellow etal. 1988). 

Comparative research has demonstrated that questionnaire 
nonrespondents can be significantly different from 
respondents with regard to attitudes and demographic 
characteristics. Gershen and McCreary (1983) found that 
individuals who are trusting and socially conforming respond 
to surveys with greater frequency than those who are not. 
Other factors which have been associated with the tendency to 
respond to questionnaires are higher economic status, higher 
levels of education, age, gender, higher academic 
achievement, and positive attitudes toward academic life and 
academic preparation (Rosenthal and Roshnaw, 1975; Ferber, 
1978; Barton et al., 1980; Hamitt and McDonald, 1982; 
Hutchinson et al., 1987; Goodfellow et al., 1988; Carifio et 
al., 1991). 

While a statistical evaluation of comparability across 
respondent and nonrespondent groups provides a strategy for 
assessing the extent and nature of nonresponse bias from all 
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sources, little is currently known about potentially biasing 
effects associated with specific sources of nonresponse. 
Included among 14 factors enumerated by Brooks (1970) as 
potentially contributing to nonresponse is undeliverable 
questionnaires due to incorrect mailing addresses, while 
this has been a persistent problem in survey research, its 
impact as a potentially biasing factor has been relatively 
unexplored. 

The circumstances surrounding the nonresponse behavior 
of reachable and unreachable nonrespondent groups appear to 
be sufficiently dissimilar to warrant the separation of the 
two groups in analyses. For one group, nonresponse behavior 
is voluntary since it is based on a conscious unwillingness 
on the part of the student to respond while for the other 
group nonresponse behavior is involuntary, the result of 
failure on the part of the College to make contact. This 
study was designed to determine if there were student 
characteristics that systematically distinguished among 
survey respondents, nonrespondents who choose not to respond, 
and nonrespondents who are unable to respond because the 
College fails to make contact with them due to inaccurate 
mailing addresses. 

Nature of Bad Address Problem 
Concern over the influence of nonresponse bias in survey 
data is germane given the large amount of institutional data 
gathered by mail surveys. The extent of the problem is 
exemplified by, although not limited to, the experiences of 
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the Office of Institutional Research which has systematically 
gathered information about the post-community college 
experiences of graduates and former students through mail 
surveys, while satisfactory response rates have generally 
been associated with graduate surveys, a persistent challenge 
has been encouraging responses from former nongraduates who 
may have a diminished sense of connection with the College 
given their short-term persistence or the length of time 
elapsed since their departure. Smith and Bers (1987) observe 
that temporal factors such as these often combine to 
exaggerate the nonresponse problem for community colleges. 

Statistical comparisons of respondent and nonrespondent 
groups to institutional surveys have generally produced mixed 
results. At times, small insignificant group differences 
based on demographic characteristics and academic experiences 
have been found, other surveys have been characterized by 
significant nonresponse bias given the underrepresentation of 
student groups defined by race and student academic ability 
(Grosset, 1990) . These past attempts to assess the level of 
nonresponse bias in mail survey data have not separated 
involuntary nonrespondents from voluntary nonrespondents. 



Method 

Data for this study was based on three follow-up surveys 
of former students who graduated or left the College prior to 
graduating. The surveys, which were conducted between 1989 



and 1991, shared a common data collection methodology. The 
questionnaires used for the three separate surveys were 
single sided documents printed on white paper. In each case, 
the mailing included a postage-paid envelope for the return 
of completed questionnaires. Questionnaire items provided 
structured response options or called for one-word responses. 
Cover letters and questionnaires were. mailed once and no 
attempt was made to encourage greater response rates through 
successive follow-up mailings. 

The replication of data collection methods was possible 
across the three surreys since they were undertaken for the 
same purpose. The College participated in a national effort 
to define community college transfer rates. Toward this end, 
three groups of former students were surveyed over a three- 
year period in order to learn about their transfer-related 
experiences. The cohort of interest in these surveys was 
defined as students who entered the College four years prior 
to the survey and earned a minimum of 12 credits at the time 
of the survey. The three entering cohorts included students 
who entered each fall between 1985 and 1987. The length of 
elapsed time since departure from the College varied within 
these targeted cohorts, ranging from one semester to 3 1/2 
years. 

Return and nonreturn rates for each of the surveys 
included in this study appear in Table 1 which is located in 
the Appendix. The bad address rate of return ranged from 7.2 
to 17.2%. Because of the skewed nature of the distributions 
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across the three respondent/nonrespondent categories, a 
weighting procedure was applied prior to data analysis. 
Random samples of students were selected from the larger 
pools of students comprising the "Received-Returned Survey" 
and the "Received-Did Not Return Survey" so as to approximate 
the size of the samples associated with the bad address 
category. As a result, each group represented about one- 
third of the data to be analyzed. 

Respondents, voluntary nonrespondents , and involuntary 
nonrespondents were compared in terms of demographic 
characteristics, entering academic abilities, academic 
engagement, and persistence. Demographic characteristics 
included measures of racial background (African American; 
Hispanic; Asian; White), gender, age at time of survey, and 
financial need (received financial aid; did not receive 
financial aid) in both the first and last semester of 
attendance. Entering academic ability was measured by scores 
on placement tests routinely administered to new students 
prior to their first semester of enrollment at the College. 
These tests included a reading (range from 1 to 45), writing 
(range from 1 to 18) , computational math (range from 1 to 
25), and an algebra (range from 1 to 15) test. A fifth 
indicator of academic ability was the number of remedial 
credits taken by the student while at the College. 

Cumulative GPA was used as an indicator of the academic 
engagement of the student and curricular type ("Select"; 
General Studies; Other) was included as a more generalized 
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measure of student integration, it was assumed that students 
in "select" curricula were most integrated into the academic 
and student life of the College, students in General Studies 
were least integrated, and the integration experiences of 
students in "other" curricula would fall somewhere in 
between . 

Measures of persistence included the number of enrolled 
semesters and the cumulative credits earned by the student at 
the College prior to departure. The 14 items that were used 
for group comparisons across the three surveys appear in 
Table 2 in the Appendix. 

Results 

The most consistent results across surveys were 
associated with persistence and academic engagement 
indicators. ANOVA tests indicated there were significant 
group differences based on CPA, credits earned, and number of 
enrolled semesters. m each of the three survey samples, 
respondents earned significantly higher CPAs than either of 
the nonrespondent groups, and voluntary nonrespondents 
typically earned lower CPAs than involuntary nonrespondents 
(See Table 3 in the Appendix) . Cumulative credits earned 
prior to departure from the College were significantly higher 
for respondents than for nonrespondents, regardless of 
voluntary or involuntary status. The pattern of group 
differences between nonrespondent groups, while consistently 
significant, varied in nature across the three surveys. In 
two of the surveys, involuntary nonrespondents amassed fewer 
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credits than voluntary respondents, while in one survey the 
opposite was true. The same pattern of response was true for 
the number of enrolled semesters prior to departure. 

Age and ethnicity were demographic characteristics for 
which significant group differences were indicated in two of 
the three surveys, older students were more likely to 
respond in two of the surveys and younger students were more 
likely to be unreachable because of an incorrect mailing 
address or to not respond by choice. The two nonresponding 
groups did not differ significantly with regard to age. 

White students were more likely to respond than students 
from the other racial categories (See Table 4 in the 
Appendix) . African American and Hispanic students were more 
likely to choose not to respond while Asian and African 
American students were slightly over-represented in the 
nonrespondent group due to incorrect mailing addresses. 

Financial aid status was a relatively unimportant factor 
in terms of group differences. In just one of the samples, 
former students who received financial aid in their first 
semester were more likely to be included among the 
involuntary nonrespondents than were students who did not 
receive aid. Non-aided students were more likely to return 
the survey than aid recipients. 

With the exception of remedial credits, other indicators 
of ability upon entry the College were unimportant in 
defining group differences. Only one significant difference 
was noted across the four placement tests and the three 
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samples. In this one case ,. respondents had significantly 
higher scores on the reading test than did sither group of 
nonrespondents . 

In two of the three survey samples, former students with 
fewer remedial credits were more likely to return 
questionnaires than were students who had enrolled in a 
larger number of remedial courses. The two nonrespondent 
groups did not differ significantly in terms of remedial 
credits. 

Discussion 

While the challenge is to eliminate or greatly reduce 
the number of nonresponses for any reason, the strategies for 
accomplishing better return rates will be different depending 
on the voluntary/ involuntary nature of the nonresponse. 
Effective strategies to encourage response from students who 
receive the survey include the following: short professional- 
looking introductory letter and questionnaire with simple and 
clearly asked questions that are presented in a logical 
sequence; inclusion of a postage-paid self -addressed envelope 
for ease of survey return; personalization of survey by use 
of name rather than a generic term such as 'student' or 
' alumnus ' ; and a second-wave follow-up survey mailed to 
nonrespondents to the initial questionnaire request. 

In an effort to eliminate incorrect addresses, students 
need to be reminded of the importance of informing the 
College of address changes, even after they dep. _-t. While 
the Office of Development and Alumni Relations provides a 
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mechanism for graduates and former nongraduating students who 

complete a sizeable number of credits (45+) to keep the 
College informed of address changes after they leave, there 
is not a centralized office to facilitate address changes for 
the large number of former students who are not targeted by 
the Alumni Relations office. Knowledge of changing addresses 
for these students is likely to be shared by a wider 
constituency. As department heads, faculty and staff 
throughout the institution become aware of address changes, 
they should notify the Registrar's Office who will follow 
through by either directly . changing the information on the 
data base or sending the student a change of information form 
for permission to revise their mailing address. Accepting 
the responsibility of informing the Registrar's Office will 
enhance the quality of student information used for 
assessment and planning by improving response rates and 
including information from students who have systematically 
been eliminated from mail survey data. In doing so, there is 
a primary benefit of serving students better and the 
secondary benefit of saving valuable institutional resources. 
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Table 1 



Rates of Return and Non-Return Associated 
With the Three Samples in Study 



Sample 



Fall 
1985 

Entrants 

Fall 
1986 

Entrants 



Cohort 
Size 



# 

1936 



1740 



Received- 
Returned 
Survey 

# % 
910 47.0 



891 51.2 



Received- 
Did Not 
Return Survey 

# % 
759 39.2 



724 41.6 



Did Not Return 
Survey Because 
of Bad Address 

# % 
267 13.8 



125 7.2 



Fall 
1987 

Entrants 



1547 



.690 44.6 



591 38.2 



266 17.2 
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Table 2 



Variable 



Test 



Summary Table of Survey Respondents 
and Nonrespondent Differences 



Fall 1985 
Entrants 



Fall 1986 
Entrants 



Fall 1987 
Entrants 







Value 


df 


Value 


df 


Value 


df 


Ethnicity 


Chi <?ct 


1 R DA* 


D 


IK Tlx 


6 


8 . 05 


6 


Gender 


Chi Set. 


0 OR 


9 


U • / 4 


z 


0 . 72 


2 


Age 


F-test 


3 ? 0* 


9 


U • / 1 


o o r\ ~t 

2 , 897 


3.84* 


2 , 795 


Financial Aid- 
First Semester- 


Chi Sa 


13 65** 


z 




2 


3.85 


2 


Financial Aid- 
Last Semester 


F-test 


1.56 


2 


2.55 


2 


2.10 


2 

< 


Reading Ability 


F-test 


1.12 


2,756 


3.71* 


2,897 


1.37 


2,795 


Writing Ability 


F-test 


0.40 


2,756 


0.30 


2,897 


1.60 


2,795 


Math Computation 


F-test 


1.08 


2,756 


0.99 


2,897 


0.20 


2,795 


Ability 
















Algebra Ability 


F-test 


0.11 


2,756 


1.16 


2,897 


0.70 


2,795 


Remedial Credits 


F-test 


0.5^ 


2,756 


3.08* 


2,897 


5.41** 


2,795 


Curricula Type 


Chi Sq. 


3.13 


4 


0.06 


4 


1.47 


4 


GPA 


F-test 


3.11* 


2,756 


18.53*** 


2,897 


7.83*** 


2,795 


Credits Earned 


F-test 


9.81*** 


2,756 


22.55*** 


2,897 


12.89*** 


2,795 


# of Enrolled 
Semesters 


F-test 


5.69*** 


2,756 


8.66*** 


2,897 


16.52*** 


2,795 



* p > .05 
** p > .01 
*** p > .001 
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Table 3 

. ( Scheffe Results 

Significant Differences in Group Means Across 
Respondent Groups and Samples 



Fall 1986 Fall 1987 



Group > 


1* 


2** 


3*** 


1 


2 


3 


1 


2 


3 


# of Enrolled 




















Semesters 


10.9 


9.6 


8.8 


7.6 


6.4 


5.8 


7.4 


5.9 


6.3 


Earned Credits 


44.4 


37.2 


30, ,2 


41.6 


33.7 


31.3 


52.3 


32.3 


37.3 


GPA 


3.1 


2.6 


2.9 


2.9 


2.5 


2.7 


3.2 


2.6 


2.8 


Remedial Credits 


NS 


NS 


NS 


6.3 


11. 1 


10. 6 


6.2 


10.4 


10.7 


Age 


33.2 


30.2 


29.4 


31.6 


28.7 


28. 1 


NS 


NS 


NS 


Reading Ability 


NS 


NS 


NS 


NS 


NS 


NS 


32 . 0 


29. 1 


28.9 



NS = No Significant Group Differences 

* Group 1 received and returned survey 

** Group 2 received and did not return survey 

*** Group 3 did not receive survey because of bad address 
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Table 4 

Significant Percentage Differences 
Across Respondent Groups and Samples 



Fall 1986 Fall 1987 





1* 


2** 


3*** 


1 


2 


3 


1 


2 


3 


Ethnicity 




















African Amer. 


NS 


NS 


NS 


24.0 


41.4 


34.6 


19.2 


43.4 


37.3 


Asian 


NS 


NS 


NS 


31.8 


30.5 


37.7 


36.6 


28.3 


35 1 


Hispanic 


NS 


NS 


NS 


18.9 


51.1 


31.0 


14. 3 


53.4 


3. .3 


White 


NS 


NS 


NS 


36.3 


29. 1 


42.0 


42.0 


26.9 


31.0 


Financial Aid 




















First Semester 




















Received Aid 


24 .4 


27.9 


47.6 


NS 


NS 


NS 


NS 


NS 


NS 


No Aid 


38.6 


37.8 


23.7 


NS 


NS 


NS 


NS 


NS 


NS 



NS - No Significant Group Differences 



* Group 1 received and returned survey 

** Group 2 received and did not return survey 

*** Group 3 did not receive survey because of bad address 
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